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FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT 
Not Applicable. 

BACKGROUND OF THE INVENTION: 

The present invention relates to the field of music. More particularly, the present 
invention relates to a display system for displaying musical compositions, either batch or 
in a real time environment, and processing and communicating user performances. 

Music is usually only available in the written form for one (or a fixed set of) 
performer/instrument types in a fixed key. Adaptations or variations of musical 
arrangements are complex and costly. Remotely located musicians are unable to 
effectively practice together. Small communities each with only a few musicians are 
limited to practicing with the few musicians they have. 
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Performers of music have many inconveniences to deal with. One such 
inconvenience deals with the composing, distribution, and utilization of music display 
presentation, traditionally sheet music. Another major problem relates to the 
inconvenience of scheduling and physical gathering of multiple musicians (including 
instrumentalists and vocalists), which when combined in their performance provide a 
musical ensemble or orchestra. For example, high school band practice requires that all 
students be available to practice at the same time at the same place (i.e., the school music 
room). However, this creates difficulties in that many students have other activities 
which conflict with band practice which is then incomplete. Additionally, when 
composing, musicians often will come up with an idea when physically not with another 
musician. 

Musicians typically work from sheet music. When composing, they write the 
notes down on paper that has a number of staffs. If the musician transposes a 
composition from one key to another, the notes are also written down on the staff paper. 
The scores for different instruments must also be generated and written down. All of the 
scores are then copied for distribution to other musicians and/or music stores. 

When performing, the sheet music must be found, for all parts to be played, 
manually distributed, manually set-up, manually handled (turn pages, etc.). There is also 
an unfulfilled need for quick access to a more comprehensive database of music for the 
performing musician, whether he is solo or part of an orchestra. Also, musicians often 
perform audience requests, and require access to sheet music for requested songs. 
Presently, there are various combinations of songs compiled in "FAKE" Books, usually 
by category (e.g., rock, country, blues, big band, etc.). This is only of limited help. 
Furthermore, the use of paper sheet music is cumbersome and inconvenient; pages often 
get damaged or lost, and indexed access is poor and slow. 

This method of composing and distributing music is inadequate when the music is 
used by a band or orchestra that requires hundreds of copies. If the conductor desires the 
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piece to be played in a different key or certain sections of the music edited to suit the 
conductor's tastes, the composition must be rewritten and the new transposed copy 
distributed to the band or orchestra. This is a very costly, time-consuming, and laborious 
task if the orchestra has a large number of members. 
5 Additionally, if the composition does not have a part for a certain instrument, the 

conductor must generate the required part from the original composition. After the score 
for the required instruments has been generated, the parts must be copied and distributed 
to the individual musicians. This, again, is a very costly and laborious task if the band has 
a large number of musicians requiring different parts. There is a need, therefore, for a 

10 more efficient way of transposing, editing, and distributing music scores. 

Over the past many years, great advances have been made in the electronic input, 
storage, and display of music. Electronic bands and orchestras are constructed using 
computers and MIDI equipment. Programs exist for personal computers (e.g., Apple 
Macintosh, DOS, and Windows machines) for an individual to use the computer for 

15 transposing music, composing music. Programs also exists for automatically inputting 
music from direct performance (such as directly from a keyboard, electronically through 
MIDI converters (such as for string instruments), via pickups and microphones, and 
sequencers, tone generators, etc.) To generate digital data and/or music notation. 

Musicians often perform both pre-planned and ad hoc compositions during the 

20 course of a performance. It would therefore be desirable to have the ability to access a 
large database of musical compositions on demand. It would also be desirable to permit 
communication and synchronization of a music presentation to multiple performing 
musicians who are playing together. It would also be desirable for a performing musician 
to have his or her performance of the music input onto an individual music workstation, 

25 and stored, and analyzed by an automated system, and/or communicated to one or more 
other networked (remote) individual music workstations.. 
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SUMMARY OF THE INVENTION: 

The present invention encompasses a musical presentation system. In one 
embodiment, the system enables one or more users to select a musical composition, and 
perform the selected musical composition at each of a plurality of individual music 
5 stands, independently capturing the performance of a respective user and communicating 
the individuals performance data to a master workstation (which can be standalone or one 
of the networked individual workstations), which combines the plurality of individual 
workstation performance data into a composited combined performance data and 
communicates said combined performance data back to all of the individual workstations 

10 wherein the individual workstations provide for audio (and/or vide and/or audiovisual) 
output representative of the combined performance data (which represents the musical 
performance inputs for all of the communicating plurality of individual workstations). 
The time between the users performing the segment of music and that same user hearing 
the audio presentation of the combined data is less than the time interval detectable by a 

1 5 human being. In a preferred embodiment, the plurality of individual workstations 
collectively provide for synchronized display presentation of a selected music 
composition's presentation, and provide for output of individual performance data 
representative of the musical performance of the user corresponding to the display 
presentation. Timing-synchronization is also provided for to permit synchronization of 

20 the display presentation of the plurality of individual workstations, and for 

synchronization by the master workstation of the plurality of individual performance 
data to construct the combined performance data. In one embodiment, the master 
workstation generates a synchronization signal that is communicated to all the individual 
workstations from the master workstation. Other synchronization structures are also 

25 equally acceptable, such as embedding timing synchronization data within the individual 
performance data. 
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In another embodiment of the present invention, a musical presentation system 
enables a user to select from one or a variety of musical compositions, control the 
adaptation of the selected composition, and distribute the edited version efficiently in a 
paperless environment. The system and process of the present invention also provides 
5 means for receiving music from a number of sources. The user selects the desired musical 
composition from the source. The desired composition is displayed and/or stored in the 
system's memory for further processing by the system prior to display and/or 
distribution. 

In accordance with one aspect of the present invention, each of the music 

1 0 workstations in an intelligent music composition communication architecture provides for 
musical information, to be distributed for a video or visual presentation of music in a 
user-friendly notation, and/or provides an audio presentation that can be selectively 
tracked and synched to the video presentation and/or tracks and synchs the displayed 
video presentation to a live performance, etc. 

15 In still another embodiment, the system includes a user input device enabling 

selection of the musical composition, and optionally, permitting any user specified editing 
desired in the composition, and, in a preferred embodiment, permitting user selection of 
parameters (such as the musical key in which the composition is to be played). The user 
can then instruct the system to transmit the newly generated music scores to one or more 

20 display subsystems (such as CRT's, LED's, LCD's, etc.), or to other systems. In the 
preferred embodiment, these displays take the form of liquid crystal displays built into 
music stand based systems, also referred to herein as display stations or workstations. 

This invention also relates to a musical presentation and/or communication 
system, and more particularly, to a system which permits musicians to view the score or 

25 other audiovisual or visual-only presentations of a musical composition, to permit the 

musician/user to perform the musical composition on the instrument of choice, as selected 
by the user, and in the key of choice, as selected by the user. 
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In accordance with one aspect of the present invention, one or more music 
workstations are provided, consisting of a video presentation means, such as a display 
screen (CRT, LCD, LED, Heads Up Display (HUD) etc.) in conjunction with a 
computer-based system which in one embodiment stores a database of songs and music 

5 which can be utilized by the musician/user of the system. 

In accordance with yet another embodiment of the present invention, there are 
provided numerous music workstations, such that different or similar instruments can 
each select a respective portion of a song to be performed, such that all musicians 
performing the same musical piece are provided musical presentation or representation in 

0 the same key to permit the playing of the music together. In a preferred embodiment, the 
system is capable of locking into a particular key (and/or instrument type) responsive to 
input variables (such as can be provided via an input device such as a microphone or 
keyboard, or voice recognition, or camera) to determine the desired key to be played in. 
In a preferred embodiment, the user can select an auto-transpose mode where all music is 

5 thereafter automatically transposed. The transposition can take place at a master 

workstation and then be communicated to the individual workstations. In one of the 
illustrated embodiments, the transposition of the music takes place locally at the music 
workstation which provides local intelligence. An original musical composition, such as 
selected from a stored database of a library of music is then transposed in accordance with 

0 transposition rules. Many options are available, and various transposition rules and 
methods are well known and documented and in use on numerous products, such as 
keyboards which have built-in capabilities for transposition of music, as well as computer 
software which provides for transposition and notation of music. 

In accordance with an alternate embodiment, the user connects to a remote 

5 database via wireless or wired communication to permit downloading to the local 

computer of those musical pieces which the user has selected for performance. The user 
music terminal, or the central computer, can receive the transposed (derivative 
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composition) version or can receive an unmodified (original musical composition) version, 
which it can then display and/or convert and transpose music as necessary for each 
instrument and/or key. 

In another alternate embodiment, a user can prearrange for downloading of selected 
5 compositions via a remote connection service, where the music user terminal can include a 
non-volatile storage memory permitting the storage and replay of all requested 
compositions. Alternatively, a central server can be provided, where multiple music end 
terminals share a single central controller computer, or each have their own computers 
which share in a central server computer or with each other in a distributed architecture. 

10 Alternatively, a non-volatile storage structure, such as a CD-ROM, can contain a 

database of music which can be selected from for transposition in accordance with the 
present invention. 

The present invention provides a music communication architecture and 
methodology that permits the synchronizing of music display presentations for multiple 

15 display stations performing the same musical composition. Bands and orchestras can be 
constructed using multiple independent music workstation display whether at one site or 
remotely distributed. In one embodiment, the music display presentations provides one 
or more of the display of the musical composition, performance by the user, and the 
output presentation for the combination of the individual performance data for a plurality 

20 of individual workstations. In one embodiment, a master workstation is responsive to the 
individual performance data from the plurality of individual performance data and 
provides means for synchronizing and compositing the individual performance data from 
the plurality of the individual workstations, and providing a presentation output (audio 
and/or video) comprising the combined virtual performance. 

25 It is a further object of the present invention to permit the comparison of a 

performer's performance parameters, such as parameter signals obtained via a microphone 
and/or camera, of the performing artist's performance or execution as compared to the 
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stored and displayed music. The comparison includes the pitch, timing, volume, and 
tempo etc. of the music (such as through audio recognition) and critique the artist's 
physical movements (e.g., proper finger position, etc.) through visual recognition. In a 
preferred embodiment, the music workstation system provides the performer and/or a 
5 conductor with a presentation performance feedback indicating the quality of the 

performance as compared to the stored or displayed music, such as any errors, where 
they occurred, etc. 

It is a further object of the present invention to provide a system whereby any 
displayed music can be transposed (e.g., to a different key, or converted to include 
1 0 different or additional different instruments and voices than that in which the sheet music 
is originally displayed). 

It is a further object of the present invention to provide automated modes of 
intelligent operation of the music workstations, and to provide responsiveness to multiple 
forms of user input. 

15 These and other aspects and attributes of the present invention will be discussed 

with reference to the following drawings and accompanying specification. 

BRIEF DESCRIPTION OF THE DRAWINGS: 

FIGS. 1 A and IB show a music presentation system in accordance with the 
20 present invention; 

FIGS. 2A-2G show flow charts of the processes in accordance with the present 
invention; 

FIG. 3 shows one embodiment of the display for the music display workstations 
and input devices in accordance with the present invention; 
25 FIG. 4 shows a shared music database and stand alone workstation embodiment in 

accordance with the present invention; 
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FIG. 5 shows a music communication system in accordance with the present 
invention; 

FIG. 6 shows a master workstation and slave workstations in accordance with the 
present invention; 

5 FIG. 7 shows an alternate embodiment of the present invention using one or more 

of the workstations coupled to a master controller and music database; 

FIG. 8 shows a marching band environment in accordance with the present 
invention; 

FIG. 9 shows a person outfitted with a sensor body suit in accordance with one 
1 0 aspect of the present invention; 

FIG. 10 shows a movement and pattern recognition system in accordance with 
one aspect of the present invention; 

FIG. 1 1A illustrates a flow chart for an alternative embodiment of the operation of 
the automated mode "A Mode" 240 of FIG. 2A, alternative to that illustrated in FIG. 2C; 
15 FIG. 1 IB illustrates the process flow for an alternative overall operation relative 

to FIG. 2A of a music composition communication workstation in accordance with the 
present invention, wherein steps 200-250 of FIG. 2A are redefined to show an overall 
operation which also encompasses a specific networked mode A Mode 5, as further 
illustrated in FIG 12; 

20 FIG. 12 illustrates the process flow for the networked virtual performance mode 

1150 of FIG. 11B; 

FIG. 13 illustrates the process flow for automated mode 6 corresponding to 
process step 1153 of FIG. 1 1A, directed to synchronizing the display to the 
performance, to resynchronize the display of the music to account for performer error; 
25 FIG. 14 illustrates a communications architecture in accordance with the present 

invention, where each of the individual music stands 1410 receives a music input 1411 
and provides performance data output 1412 coupled to and communicating with a master 
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workstation which generates an output of virtual performance data 1451 responsive to 
compositing the plurality of individual performance data signals which virtual 
performance data 1451 is communicated back to all of the plurality of individual 
workstations 1410 which provide audio output responsive thereto; 
5 FIG. 15 illustrates one example of data flow communication architecture for a 

plurality of physically separate remotely located locations, such as a teacher/master 
conductor/server location and each of a plurality of locations that provide communication 
of performance data for users of the individual workstations at each of those locations, 
wherein timing information is utilized by the master workstation to reconstruct and 
10 combine all of the individual performance data into a combined virtual performance data 
output coupled back to the plurality of individuals workstations which responsive 
provide audio output; 

FIG. 16 illustrates a data word structure compatible with one embodiment of the 
present invention; 

1 5 FIG. . 1 7 illustrates a block diagram for a workstation (generally shown in FIGS. 

5-7) and further comprising a network interface and a MIDI performance data processor; 

FIG. 18 illustrates a timing diagram showing the synchronization timing of the 
architecture as illustrated in FIGS. 14 and 15, and data structure illustrated in FIG. 16, 
and showing the composited synchronized virtual performance data and the multiple 
20 individual performance data; and 

FIG. 19 illustrates one example of a memory work space structure for the 
individual and master workstations in the virtual performance mode, in accordance with 
the present invention. 



25 DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT: 

While this invention is susceptible of embodiment in many different forms, there 
is shown in the drawing, and will be described herein in detail, specific embodiments 
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thereof with the understanding that the present disclosure is to be considered as an 
exemplification of the principles of the invention and is not intended to limit the invention 
to the specific embodiments illustrated. 

In accordance with the teachings of the present invention, a system and 
5 methodology are provided for music presentation and communication. Musical 

compositions can be input to the present invention from any one or more of multiple 
sources, such as from prestored score images, live microphone, direct input from musical 
instruments or vocal direct performances, scanning in of existing printed score images 
(optically character recognized), cameras, visuals, etc. These inputs by the system are 

10 used in the selective storage, composition, communication, and presentation of the 

musical system of the present invention. The system can generate additional material 
automatically, or permit a user to modify, communicate, display and/or reproduce the 
musical compositions. 

Modification can be performed on rhythm, primary key, individual notes, chords, 

15 etc. The vast store of musical information stored in digital notation format and/or any 

video format, can be broadcast (analog or digital) to a local music workstation or a master 
controller, which can also be a local workstation. The master controller can be a stand 
alone unit, or act as a server as well as its own stand alone unit, or simply as a server to a 
plurality of other stand alone units. However, in the minimal configuration, only a single 

20 musical user station is needed. 

In one preferred embodiment, the workstation is provided as a music stand where 
the display presentation is a liquid crystal display (LCD). The LCD that can provide 
monochrome, gray scale or high quality color displays, depending on design and cost 
constraints and desires. Other display types can also be used. A touch-screen input can 

25 provide for simplified and adaptive user input selections. An optional built-in 

metronome function can also be provided for display presentation audio and/or video. A 
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subsystem can also be optionally provided to permit the music to be audibly reproduced 
at the workstation through a speaker or headphone jack or other output. 

It is well known in the art to convert user analog audio input into a digital format, 
ranging from straight Analog to Digital (e.g., A/D) conversion to processed data 
5 conversion to encoded digital music data, such as MIDI. Examples of MIDI include 
guitar input or other stringed instrument input through microphones or directly to 
MIDI-converters, or voice/non-pickup instruments through microphone converted to 
MIDI-input, or keyboard MIDI-input. Such input systems are commercially available 
from numerous companies for numerous types of interfaces at numerous interface levels. 
10 Similarly, numerous A/D converter subsystems are commercially available at chip and 
board solution levels (such as from Analog Devices Corporation and from Mattrox 
Systems). 

In accordance with one aspect of the present invention, the multi-dimensional 
music transformation system of the present invention also enables a user to select one or 

1 5 more musical compositions from a larger database from a plurality of musical 

compositions. The database can be stored locally within the workstation, on site, or 
remotely stored and transmitted to the user (such as over cable, wire, telephone lines, 
wireless (such as radio frequencies)). The user can also optionally edit the selected score 
of the composition (such as changing the key and/or any note and/or timing, etc.) to suit 

20 his or her taste. The score (altered (the derivative composition) or not (the original 

composition)) can then be transmitted to one or more displays such as liquid crystal or 
CRTs in the music stands of the band or orchestra. The present invention, therefore, 
provides an efficient, paperless solution to communicating, presenting (displaying), and 
optionally one or more of transposing, editing, inputting, comparative testing-teaching, 

25 conducting, and disseminating music to one display or a large number of displays. Each 
display can have the same, or a different, unique, customized presentation of music 
notation as appropriate per selection, responsive to a set-up by a system, automatically 
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per predefined parameters, and/or to user input. The score can also be printed out if a 
hard copy is desired. 

As illustrated in FIG. 1, the music is stored, such as on a large hard drive or CD 
ROM jukebox, in a digital format as a music library (120). The music library (120) is 
5 coupled to a processor subsystem (115). Coupling can be wireless or cabled such as 
through a shielded cable, fiber optic conductor, switched connection (such as via phone 
lines), local, or remote. The processor (115) has the local storage capacity (e.g., semi- 
conductor memory, disk storage, etc.) to hold the digitized version of the music 
composition transmitted to it on request from the libraiy (120). The music library can be 

10 local or proximately remote from the rest of the system. 

In a wireless embodiment, the music library (120) is coupled to a communications 
subsystem (such as a radio frequency transmitter) (125) that transmits the contents of 
requested compositions from the remote music library (120) to the processor (115) 
through the antenna (104) of the transmitter. The antenna (102) at the receiver picks up 

1 5 the transmitted signal and the receiver conveys it to the processor (115). This 

embodiment enables the music library (120) to be remote and located at a great distance 
from the requesting site. The communications subsystem (125) can be a transceiver for 
bidirectional wireless communication, or a transmitter for one-way wireless 
communication (such as where the requests are otherwise communicated to the music 

20 library subsystem (120), such as via a wired connection). 

As illustrated in FIG. 1 A, a system controller, in the form of a music stand 
(105C) with a liquid crystal display, is used by an operator (e.g., performer, conductor, 
etc.) to select one or more musical compositions. FIG. 1A illustrates two types of music 
workstations stands. The workstation stand (105C) provides certain optional features 

25 for a more full-featured stand, including as illustrated, speakers (140) both wireless and 
wired communications capability, and as illustrated, shows the processor with memory 
(115) as an external separate component. The music stand (105P) shows the integration 
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of the processor and memory into the music stand itself, and also shows both wireless 
(antenna (101)) and wired connection (port (107)) to permit network communication. 
Alternatively, the conductor stand (105C) could have all or part of the features integrated 
into the music stand (105C). Depending on the function for which the music workstation 
5 stand will be used, some or all of the features can be provided for that stand to minimize 
costs or optimize versatility. For example, in one situation, only the teacher or conductor 
needs the full-featured, full-powered music workstation. In that case, the performers or 
students do not have a full-feature workstation, but rather a scaled-down version of the 
workstation stand. In the preferred embodiment, a user input device (110) (such as a 

10 touch screen, microphone, keyboard, switches, voice recognition system, visual 

recognition system, etc.) is coupled to the processor in a wired (such as over a cable or 
fiber optic link) or wireless (such as over an RF link or infrared link) manner for 
workstation stand (105C), or directly to the processor, where it is built into the system 
controller as workstation (105P). The user can select an original musical composition 

15 from the touch screen of the liquid crystal display (135). The processor responds by 
storing that composition in the memory (115) of the local workstation of the user as 
requested. 

Using the touch sensitive LCD (135), the user can now create a derivative musical 
composition. The touch sensitive LCD allows the user to enter the musical key in which 

20 the original composition will be played, edit any notes desired, and select the instruments 
and parts that will be playing the composition. The composition as originally composed, 
and the derivative or modified composition can be played back to the user over speakers 
(140) so that he or she may listen (e.g., such as to observe how the changes will sound) 
while optionally permitting simultaneous viewing of the score on the presentation visual 

25 display. Once the score has been designated (e.g., selected, edited, etc.) to the users (e.g., 
conductor's) taste, the appropriate portions (e.g., by musical instrument) of the scores 
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can then be transmitted for (optional storage and) display to the respective associated 
individual music workstation stands of the band members. 

In a preferred embodiment, each stand has an input device (110) that permits the 
user of the stand to select which instrument will be using the stand. (As discussed above, 
5 this input device can take the form of a touch sensitive screen or a number of buttons or 
switches or voice or audio recognition, etc.) 

In the preferred embodiment, each individual music workstation stand (105) can 
be directly and/or remotely programmed to addressably receive (and optionally to locally 
convert) and display the music score that is intended for the respective instrument type 

10 (user type) that will be using (is associated with) the stand. As an example, the user of 
the stand (or a conductor) can input their selection of saxophone into the user input 
device (110) of the workstation stand (105C), to program that workstation stand (105C) 
only to receive the musical score for the saxophone (see FIG. 3). Then, the musical 
scores for all selected parts can be independently broadcast to all connected workstation 

15 stands, with each individual workstation stand individually distinguishing and accepting 
only its part. Alternatively, each workstation stand can be individually addressed for 
separate broadcast reception of its own respective selected part. Additionally, the user of 
the stand can program the user input to select a musical part of a selected musical 
composition (e.g., saxophone first chair) and receive only the musical score intended for 

20 that chair. This same procedure can be followed for other instruments within the band or 
orchestra. Alternatively, a single music composition can be broadcast to all workstations, 
where each workstation has local intelligence (processing and storage) to permit local 
conversion for display at each workstation for the selected instrument for each 
workstation. 

25 For wireless communications, the individual music workstation stands (105) are 

comprised of receivers (or transceivers where bidirectional communication is desired) and 
antennas (101, 103) for receiving (or transceiving) the radio frequency information from 
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(and to) the master workstation (such as for the conductor). The music stand also has a 
display (such as an LCD (135)) for displaying the musical score intended for that stand. 

Referring to FIG. IB, the music workstation stands can either be identical or 
broken down into conductor stands and performer stands. A conductor stand (105CON) 
5 may have more functions and control than a performer stand (105PER). A performer 
stand (105PER) might only have the ability to receive and display musical scores, 
whereas the conductor stand (105CON) has the ability to select the musical score, change 
the key of the musical composition, and perform other tasks only a conductor would be 
permitted or required to do. 

10 In one embodiment, an RF antenna for the stand (105) can be built into the stand 

itself. Alternatively, instead of using RF, the performer's stand can be linked to the main 
(e.g., conductor's) stand using infrared, fiber optic cable, shielded cable, or other data 
transmission technologies. As discussed above, the communications link can be 
bidirectional, such as to facilitate requests and responses to facilitate the feedback of 

15 performance parameters or such that any workstation can be a master or slave, or used in 
combinations. 

FIG. 2A illustrates the overall operation of the music composition communication 
workstation. It begins by starting up the system (200). The system then provides a 
menu (201) that allows the user to select a listing of available music compositions. The 

20 user then selects one or more compositions (210). If the user selects one from the menu 
that is locally stored, it directly retrieves the information. Alternatively, if it's not 
something locally stored, the system couples (e.g. will dial up or go through a database or 
network) to a remote storage site and requests and receives the selected compositions. 
Any changes that are desired to the composition can be selected at the next logic 

25 block (215). If there are changes (such as to the key, or note editing, or selection of form 
of display or instruments), then those can be accomplished as illustrated at blocks (255) 
to (285). 
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If no changes are desired, the musical score for the composition that is selected is 
broadcast, transmitted, or otherwise transferred to the workstation music stand (220). It 
is internally stored in the local workstation music stand. Next, the score is displayed 
(225) on the workstation display (e.g., LCD or CRT) or a video projection system. The 
5 display can also be part of an integral stand-alone workstation or an interconnected group 
of components including a personal computer (such as Macintosh, or DOS or Windows 
PC). 

The display mode selection is then made (230). This permits selection of an 
operational display mode, not simply choosing the resolution or color. The two main 

1 0 choices in the preferred embodiment are a manual mode (250) and an automated mode 

(240). In the automated mode selection (240), there are many sub-modes or options, such 
as the operational mode that permits the performer or user to do their performing without 
having to tend to the selection of the portion of the music to be displayed or the turning 
of pages. In the auto performance mode as shown on LCD (135P), there is provided the 

15 simultaneous displaying of the selected musical composition, and a display representative 
of the audio performance of the user, and a divergence signal or divergence data 
representative of analyzing the performance, preferably in approximately real-time. 

FIG. 2B illustrates the manual mode (250), which provides for user manual 
selection of functions (252). There are many functions that the user can select, even in 

20 the manual mode, such as hitting a button or a touch screen to cause the turning of the 
page of the display. Another function is to go back a page or to scroll forwards or 
backwards. For those who are vision impaired, another function can increase the font size 
of the music presentation. 

Thus, there are many manually selected functions that can be provided. While the 

25 manual mode can have automated functions selected, it is distinguished from the 

automated mode where control is partially predefined without user assistance. In the 
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manual mode (250), the user selects any and all features that are going to be provided 
(some of which can be automated). The selected function is then processed (256). 

Next, any ongoing needs are processed (257). These needs can include any 
overlapping automated function (not otherwise inconsistent with any other selected 
5 function). 

Referring to FIG. 2C, the operation of the automated mode "A Mode" (240) is 
illustrated. First, the user selection of the desired automatic mode is detected and 
responded to, illustrated as the auto-advance mode (242), the training mode (244), the 
performance mode (246), or any one of a number of other modes (248) as is described in 

10 further detail hereinafter. For example, auto repeat mode can be selected by designating 
the start and stop points, and the number of times to repeat a "looped" portion (or 
portions) of the displayed musical composition. Marching band mode (auto-advance 
based on metronome function, conductor control, etc), auto-compose mode, and many 
others can also be implemented. The order of selection of auto-advance, training, or 

1 5 performance mode is arbitrary, and the user can alternatively decide from a menu where 
all are simultaneously presented as choices. 

The display can advance the music by page option, or by a user selection of one 
of many options (e.g., scrolling, tablature, video graphic tutorial display, etc.). 

Referring to FIG. 2D, the automated mode 1 for auto-advance operation (242) of 

20 FIG. 2C is illustrated, where the user has selected an auto-advance performance mode. In 
this mode U A Mode 1" (271), the system tracks the performance by the user of the 
composition to the score (272). Performance refers to the actual performance by an 
individual person (or people) who is (are) reading the musical score upon which the 
performance is based. Whether that score is in tablature format, staff and clef and note 

25 notation, or some other format, the system generates appropriate signals to permit 
comparison of the user's performance to the musical score. 
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Based on a comparison, a decision is made pursuant to selection criteria 
programmed into the system (such as the rate at which the piece is being played, the time 
signature, the tempo, the rhythm, and the advancement of the music on the available 
display), the display presentation is advanced (274 and 278). In some cases, the music 
5 might move backwards, such as with D.S. Coda. The presentation of the display tracks 
the performance to permit smooth, uninterrupted playing or singing. The capability can 
be provided for the user to override this auto-advance, such as for practicing where it is 
desired to keep going back over sections. In this case, a user override option (276) is 
permitted to alter the automated operation. Upon cessation of user override, the system 

10 can be programmed to stop, to automatically return to the regular auto-advance mode, or 
to process other auto-modes (270) of FIG. 2C. 

Referring to FIG. 2E, the automated mode "A Mode 2" (244) operation of FIG. 
2C is illustrated corresponding to the training mode. In this mode, the system tracks the 
performance (280) of the individual user to the composition score, primarily for the 

1 5 purpose of permitting a critical analysis and comparison of the performance to the score 
(282). This analysis determines divergence from the selected musical score, and reveals 
errors or deviations from desired performance goals (e.g. match of timing of notes, 
duration of notes, pitch of notes, etc.), and to display those errors (284) (such as by 
audio or video means). Predefined performance goals provide the knowledge basis for 

20 expert system based analysis. 

The system can then generate a graded score (286) indicating errors, and can 
present it in numerous formats such as histograms, frequency of errors, spacing of errors, 
etc. Identification of when the errors occur (e.g., only when going from slow to fast, or 
fast to slow), absolute position within the score and so forth, are also tracked and 

25 reported. Other expert system rules can be provided by music teachers which give the 
necessary parameters for modeling expert system reasoning, as well as guidance and 
suggestions on how to correct problems such as via display text, graphics, audio, etc. 
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The comparison of the performance to the score in the training mode is for the 
purpose of detecting the performer's compliance to parameters (such as the tempo, 
rhythm, filter, parameter, pitch, tonality, and other features that are adaptable or can be 
modified by performers). This parameter information is available and published in 
5 numerous forms. Thus, having provided this core set of parameters, the system can 
thereafter perform the training automated mode. 

As illustrated in FIG. 2F, automated mode 3 "A Mode 3" is the performance 
mode (246). In this mode, the operation is as in automated mode 1 (auto-advance mode) 
except that no user override is permitted. Its primary purpose is to accompany the 

10 performer during the entire performance of a score as an automated page turner. The 
tracking of the "page turning" to the performance can optionally be based on inputs or 
criteria independent of a performer's actual performance input (e.g., microphone), such as 
a built-in metronome clock, a central control (e.g., a conductor or special user input), etc. 
Additionally, performance characteristics can be tracked, computed, and reported as in 

15 the teaching and training mode. Training feedback can optionally be provided real-time, or 
subsequent to completion of performance, to assist the performer as in the training mode. 
Alternatively, the score can be presented in a moving score mode (e.g., vertically, 
horizontally, or otherwise) or linear presentation as opposed to a real page turning 
display. 

20 FIG. 2G shows the operation of automated mode 4 ("A Mode 4") which provides 

for the processing of other automated functions selected by the system. These modes can 
include conductor mode, karaoki mode, etc.. 

In conductor mode, a conductor can control communications of signals to his or 
her performer (such as "increase volume", or "increase tempo", or "play staccato"). 

25 Icons can be provided where the conductor simply touches a touch screen (or other input 
mechanisms) to supplement his hand and body motions to permit more effective 
communication with the performers. Alternatively, as illustrated in FIGS. 9 and 10, in a 
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more advanced system version, the conductor's movements are first learned by a 
monitoring system, based on user definition and assignment of meanings for movement to 
provide an expert knowledge database. 

This system provides for tracking of movement input such as in FIG. 10 via video 
5 camera (1005) input of the conductor (1015) against a backdrop (e.g., blue screen) (1010) 
is processed by video processing unit (1020), or, as shown in FIG. 9, via body glove 
technology (gloves (935) or sensors (944) or sensor clothing (940) or head or eye 
movement tracking sensor (930) (such as used in virtual reality, flight simulation, avionics 
equipments (such as jets and space travel), and sports players for analyzing movement) 

10 to provide the necessary movement input. This movement input is analyzed utilizing the 
expert knowledge database to automatically generate a display (video and/or audio) to 
provide local visual and/or audio reinforcement on the local display (such as overlaying on 
a then unused portion of the music score display as a picture in a picture) to permit audio 
and video reinforcement of the conductor's body language. Thus, "a hush" body language 

15 signal that is directed towards a particular section of the orchestra would automatically be 
interpreted to cause the system to indicate, and only on that particular section's 
respective displays, a message (e.g., big face with a finger in front of it making a hush 
sound with a "hush" sound simultaneously output from a speaker). The conductor mode 
provides many benefits to performance and communication. 

20 For all these automated modes (e.g., A Modes 1, 2, 3, 4), training feedback can be 

provided real time or subsequent to performance at either or both of the performer's 
workstation and a second (e.g., teacher's) workstation. 

The advantages of electronic music composition, communication and display are 
many. In addition to those discussed elsewhere herein, a capability exists for expert 

25 system based artificial intelligent type assistance where the expert system assists in many 
of the functions performed in musical composition and performance. For example, in the 
Auto-Compose Mode, if the words need to be changed to match the meter, equivalent 
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terms can be chosen from the many sources such as thesaurus, dictionaries, rhyming 
dictionaries, encyclopedias, etc., to assist as well. Phrases from poetry, selected and 
indexed by content or topic can be re-expressed to create new works. Drum and rhythm 
section accompaniment can be expertly suggested, as well as harmonies, melody lines to 
5 accompany chords, chord progressions to accompany melodies, harmonies to accompany 
a melody, and suggested musical instrument groupings to support a particular sound, 
rhythm, style, tonal quality, etc. 

The expert system can be built from commercially available technology, including 
component hardware systems with supporting software, as well as commercially 

10 available software packages which operate on commodity-type personal and business 
computers such as the Macintosh by Apple Computer, Windows and DOS machines 
based on the X86 and Pentium processor technology of Intel, technology based on the 
Power PC and 68XXX processor by Motorola, DEC PDP-11 technology, Sun 
workstations, etc. Custom microcomputer or DSP based system architecture on a chip 

15 can also be constructed, as well as ASICs, custom or semi-custom logic. 

The system can be designed to take advantage of expert system design knowledge. 
A database of rules and facts are provided, and accumulated over time by the system in a 
self-learn mode. The expert system itself has the necessary logic to probe the user, 
monitor the performance, and apply the rules to provide feedback and reports to the user 

20 of skill level, errors, automated performance display, etc., starting with a base defined set 
of rules, instructions, and a knowledge database specific to music. 

The form of the musical score communication can be easily shaped to fit needs. 
One example is MIDI (Musical Instrument Digital Interface standard) which has 
advantages such as of bandwidth of storage used, is widely available commercially, is 

25 standardized, etc. However, signal processing, text, icon-based, object based, and various 
other forms of storage, user interface, and processing can also be applied to more specific 
applications of product. 
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FIG. 3 illustrates one embodiment of an LCD display used for input control and 
for displaying the information from the processor and memory. In the preferred 
embodiment, this LCD is a touch sensitive screen enabling the functions associated with 
each displayed button to change, and also for the displayed buttons to be moved around 
5 the screen, depending on the function to be activated. The musical score may be edited 

by the conductor, such as by touching the individual note after which he is presented with 
a number of notes to replace the touched note. The lower portion of the screen displays 
instruments from which the conductor can select which instrument will be playing the 
composition. After a button on this screen has been touched, a number of sub-screens 

10 may come up, each with their own individual touch sensitive areas and functions to be 
activated by those areas. Alternatively, in addition to or instead of the touch screen, the 
system can provide input via separate key switches, voice recognition, etc. 

As an example, if the conductor touches the transmit key on the main screen, he 
will be presented with a screen showing all of the instruments that he has selected for that 

1 5 piece and a button labeled "ALL". He may now transmit to each individual music stand 
or by depressing the "ALL" area, transmit to the entire orchestra. 

The music library can be contained ("stored") on non-volatile storage either locally 
or at a remote central site containing the entire (or a subset) database of all possible music 
(that is then downloaded to local storage on request, either real-time at performance time 

20 or in advance.) 

Alternatively, the music library can be provided on storage medium that can be 
easily transported and used on site locally with the presentation system. Thus, for 
example, disk drives, cartridges, FLASH RAM cards, plug-in memory modules, or a 
CD-ROM or multiple CD-ROMs in a CD-ROM changer can be used to store and contain 

25 massive data libraries on musical compositions. While this would be a more expensive 
route than shared use of a central library, requiring each musical group to obtain libraries 
on all possible compositions they may want, it has the advantage of speed, flexibility, no 
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need for communication with a separate remote source, and creates a whole new mass 
marketing area (such as for CDs or Digital Audio Tape (DATs)). Another way of 
utilizing this technology is to maintain a history of music used, either with the remote 
music library or local music library. This could be done for many reasons, including 
5 copyright royalty assessment, determining a history of musical performances and 

requests for future use in determining performance itineraries, etc. Alternatively, a hybrid 
of locally stored and centrally shared libraries can be utilized to optimize cost, speed and 
flexibility benefits. 

In accordance with another aspect of the present invention, each display 

10 workstation can also provide the ability to convert performed musical compositions into 
annotated musical compositions, generating the appropriate musical notation (e.g., staff, 
tablature, MIDI), notes, time signature, key, instrument, or user type, etc. 

The display workstation can be implemented as a totally self-contained 
workstation, where each workstation contains its own processing subsystem, optional 

1 5 communications interface (such as wireless or cable) for network use, input/output 
interface including one or more of a user input keypad, a speaker, a microphone, 
joysticks, push buttons, etc. Each of the stand alone workstations can then operate with 
a local database or couple to a shared music database as illustrated in FIG. 4. 

The stand alone workstation(s) (105), are coupled to the shared database interface 

20 (405), and can either couple remotely (e.g., via phone lines) to the remote shared music 
database or to a local shared or dedicated music database (410). The shared music 
database (410) can either be primarily a storage means (e.g., hard disk or CD-ROM), or 
can include a processing subsystem (420) for local intelligence. In one embodiment, the 
stand alone music workstation includes the shared music database (410) and interface 

25 (405), non-volatile local storage medium for the shared databases (410), and a local 

processing subsystem (420), and can operate completely stand-alone. In an alternate 
embodiment of this stand-alone device, the shared database interface is contained in the 
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stand-alone workstation (but not the shared music database or processing subsystem), 
and provides capability for communication with a stored database (410) remote from the 
stand-alone device. 

In either of these embodiments, an alternate additional embodiment provides 
5 capability for each stand-alone workstation to function as a master stand-alone, or a 
master or slave workstation within a workstation set including multiple stand-alone 
workstations, wherein one is designated master and the rest are designated slaves. The 
slave workstations in this configuration receive communication of music compositions to 
be displayed from the master workstation, thereby permitting one shared music database 
10 to be communicated among all workstations which are a part of the group. It is to be 

appreciated that the shared music database function can be distributed in many different 
ways among the workstations, or separable from and independent from the workstations. 
The choice is simply one of design, and the illustration herein should not be taken in a 
limiting manner. 

15 In one embodiment, the master workstation has complete control over the slave 

workstation. Anything displayed on the master workstation is also displayed on the 
slave workstation. It is also possible for the user to mask certain portions of the display 
of the master workstation before it is displayed on the slave workstation. In this manner, 
the conductor, using the master workstation, can transmit to the slave workstations only 

20 that information that is required by the orchestra members. 

In an alternate embodiment, the slave workstation communicates performance 
parameters or deviation signals to the master workstation, for error analysis feedback. 

In accordance with another aspect of the present invention, means are provided 
wherein a plurality of individual workstations are coupled together in the network 

25 configuration to provide for networked communication of musical performance data 

wherein each of the individual music workstations provides for capturing the performance 
data for a users performance and communicating that performance data to a master or a 
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conductor workstation which synchronizes and combines the plurality of individual 
workstations performance data to create a composite virtual performance data output 
which is recommunicated back to all of the individual workstations in approximately real 
time, so that the individual workstations can receive the composite virtual performance 
5 data and provide an audio output (and/or visual presentation) of the combined composite 
virtual performance data including all of the individual workstations users performances. 
While the networking can be used in conjunction with other features and embodiments of 
the present invention, including communication of musical compositions for display, and 
other musical performance data analysis and data communication, as well as 

10 inter-musician interpersonal communication, the networked embodiment of the present 

invention permits synchronized virtual performance thereby permitting multiple remotely 
located individual workstations to physically separately perform with the benefit of 
hearing in approximately real time the combined result of all the performances at the 
individual workstations with their own individual performance. 

15 In accordance with another aspect of the present invention, means are provided to 

permit a user of the music workstation to accomplish a transposition of a musical 
composition in pitch, tempo, and otherwise. In a preferred embodiment, the lead voice or 
instrument can audibly indicate the key via the microphone input or via another type of 
input stimulus. The workstation can analyze the user input, determine the key, pitch and 

20 tempo for a musical composition being partially performed by the user, and adjust and 
transform the composition to be displayed in the new user desired key, pitch, tempo, 
etc., either solely for use on that workstation, or communication for use on one or more 
other workstations. In a networked version, this user input can also be communicated to 
other workstations for use by one or more of the workstations in transposing, or 

25 communicated to a master workstation, which transposes and rebroadcasts the 
transposed composition. 
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Alternatively, the user can input the pitch, tempo, and key via the user input (e.g. 
keypad, joystick, push buttons, voice recognition, playing of an instrument, etc.) and the 
system performs the transformation and displays (and/or prints out and/or audibly 
performs) the modified transformed composition for the user. Additionally, where a 
5 musical composition is written for one instrument and a different or additional instrument 
version is desired for simultaneous performance, the user can indicate the other 
instruments via the user input, and the system will generate the appropriate displays. 
The workstation can also provide an audio output of the transformed musical 
composition, either for the individual additional instrument or voice transform and 
10 present it, or for the composite of additional versions and the original version, to hear the 
blended piece. 

Referring to FIG. 5, a music communication system is illustrated comprising 
multiple workstations (500) each comprising a display (510), user input such as a keypad 
(522), a joystick (524), push buttons (525 & 526), a microphone (527), and a speaker 
1 5 (528), The workstation also includes communication interface means such as a wireless 
interface including an antenna (531), or alternatively or additionally a wired or cabled 
communication interface (540). Each workstation further includes a local microcomputer 
subsystem (550) that provides local intelligence and management of functions in the 
workstation. 

20 In the networked embodiment, where multiple physically separate locations each 

having one or more individual workstations provide for communication of performance 
data from the individual workstations and presentation by the individual workstations of 
the combined virtual performance data at the individual workstation, to provide for a 
virtual performance. In this case, the communications interface would utilize slightly 

25 different structure, such as a phone modem (analog modem), a cable modem, ISDN as 
between locations, etc. 
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Communications interfaces of various types are well known and commercially 
available. At the present time, they are available for purchase at the chip, board, or 
system level. In fact, many single chip microcomputers include communications interface 
capabilities, wired or wireless. 
5 The workstation further includes an optional musical instrument input (562) and a 

musical instrument output (564) that permit the coupling of a musical instrument via a 
musical instrument interface (570) directly to the workstation. Thus, a keyboard, electric 
guitar through appropriate input, or a microphone input through the interface (570) 
permits instruments or voices to be directly input to the workstation for direct input 

1 0 independent of the microphone (527). 

The instrument output permits coupling of the instrument input signal, either 
directly fed through or as modified by the workstation for output to the appropriate 
public address or amplification and presentation system or separate analysis system. 
The workstations are coupled either via wired or wireless communication to a processor 

15 subsystem (580) that includes a processor, non-volatile memory, read/write memory and 
an interface to a non-volatile storage medium (582). 

The processor subsystem (580) includes an appropriate communications 
interface, such as a communications interface (540) for wired interface or (532) for 
wireless interface including antenna (533). The processor subsystem couples to a 

20 non-volatile storage medium (582) containing, among other things, application programs, 
transformation programs, and either a shared music library interface application program 
or the actual shared music library and access program. 

As described above, the processor subsystem (580) and non-volatile storage (582) 
music library can be built directly into one of the music workstations (500) to be a 

25 master, with the other workstations being slaves, that can either include the processor 
subsystem and non-volatile storage or can be lower cost dummy slave terminals. As 
illustrated in FIG. 6, a first master workstation (300) provides a basic workstation 
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subsystem (200) plus contains the processor subsystem (280) and non-volatile storage 
system (285) as a part thereof so as to provide a complete stand alone music 
communication system, and be capable of acting as a master or master/slave. This master 
workstation(s) (300) can function as a stand alone, or can couple to one or more other 
5 workstations, including one or more masters (300) and/or one or more non-master 
workstations (105). 

The multiple connected workstations can operate as stand alone workstations 
using their local intelligence for displaying downloaded or resident music compositions. 
They can also interact in a master/slave linked environment, where one of the master 

10 workstations (300) asserts a master status, and all other interconnected workstations, 
whether workstations (105) or master/slave workstations (300) operate in a slave mode 
coupled to independent on the designated master. Additionally, masters can communicate 
between each other for a master/master network configuration. 

Alternatively, the multiple connected workstations can operate together in a 

1 5 networked virtual performance mode, or in a networked communications mode. A 

dedicated stand alone or distributed master workstation architecture can provide for the 
coordination and combination and synchronization of the multiple individual performance 
data outputs into a combined virtual performance data output. 

Referring to FIG. 7, an alternate embodiment of the present invention is provided 

20 where one or more workstations (105) include, at a minimum, a display of the music 

notation. These workstations are coupled to a master music communications controller 
(415) that provides for a separate user input (411) which provides input interface, such 
as to a MIDI status stream, computer data links (such as RS232, modem data link) etc. 
that designate requested musical compositions, transformations, and display requests for 

25 various ones of the coupled workstations. 

In an alternative embodiment, the master music communications controller 415 
provides for additional functionality including virtual performance mode, wherein the 
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input interface (such as the MIDI stream, computer data links, etc.) provide one or more 
of musical compositions data for display, transformation information, display requests, 
user individual performance data, and wherein the workstations respond to the master 
music communications controller to couple their individual performance data and receive 
5 back the combined virtual performance data. 

The workstations (105) access the music database storage means (420) that 
provides the data for the requested music composition via the master controller (415). 
The master controller (415) displays both the requested music composition as well as 
user interface communication for the music communication system to be displayed on 

10 either a dedicated display (416) or on one of the workstations (105) as designated by the 
master controller (415). The music database (420) can either be local, or can be via a data 
link (e.g., phone line, RF, otherwise). In one embodiment, a motion sensor subsystem 
(422) monitors motion of a target person and responds in accordance with predefined 
movement interpretation characteristics parameters, such as for a conductor. 

15 In a preferred embodiment, the user input means (411) is comprised of a key 

switch device, such as a touch membrane keypad or capacitance touch surface. 
Alternatively, in one preferred embodiment, the user input is provided via a touch screen 
technology. Touch screen technology permits the display of user interactive icons and 
legends including text and graphics making possible unlimited customization of user input 

20 structure according to task needs. Thus, specific switches or sequences of touches to the 
touch screen can be associated with common use icons from the task being performed in 
conjunction with words to provide ultimate clarity. User error is virtually eliminated, 
with the aid of automatic entry error detection, such as defined fields, mandatory fields, 
etc. 

25 Alternatively, the microphone input (527) can provide for coupling of user speech 

to a processor subsystem (580) that uses any of a number of commercially available and 
well known speech recognition algorithms. These algorithms provide for speech 
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recognition input control, either solely or as a supplement to touch screen or other tactile 
input mechanisms. 

In a deluxe embodiment, an output (421) is provided that permits coupling of an 
external display, such as a color monitor, projection unit, or other display presentation 
5 system including one or more of audio, visual, and audiovisual. 

Additionally, the display presentation output of the workstation can provide for 
an audio output presentation of the musical performance, either generated by the work 
station responsive to the music composition data, to the users performance, to the 
combined virtual performance data, or responsive to an external source. Additionally, a 
10 visual or audio visual presentation can be provided to provide information feedback to the 
user on both their individual performance as well as from and between individual 
workstations and/or the master controller or conductor workstation. 

In accordance with another aspect of the present invention, means are provided 
for moving through the printed (displayed) notation of the music in synchronization with 
1 5 the live performance from the displayed musical notation. 

Musical notation is used, in the generic sense, to refer to any way of conveying 
musical performance instructions including but not limited to common musical notation 
with staffs, notes, sharps, flats, and clefs, extending to written instructions in text form to 
supplement this or supplant or partially replace, as well as alternate forms of expression 
20 such as chord charts, words and chords (letters), tablature, any video, graphic, audio, 

audiovisual or other display presentation or combination of the aforementioned types of 
presentations. 

An annoyance in performing music using any written notation (whether on paper 
or displayed on a presentation apparatus such as a screen) is smoothly performing music 
25 and timing "flips of pages" (or the communication of change to a third party, such as 
touching a key or button). This is especially true when the user is marching and both 
hands are required simultaneously. 
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In accordance with one aspect of the present invention, means are provided to 
accept inputs from one or more sources that initiates a "page turn". Types of inputs 
include conventional touch input apparatus (such as key switches or capacitive touch 
pads), motion sensing gear, and automatically when operating in the operational mode of 
5 Auto Mode. The motion sensing gear can be for a portion of the performer's body, such 
as a head tilt sensor or an optical eye movement sensor, etc. 

Additional types of inputs that can initiate a "page turn" include voice or sound 
recognition apparatus built into the microcontroller system. This apparatus has the 
ability to use pattern recognition specific to the sound or user voice and words being said 

10 (for extremely high accuracy). Of course, any type of user actuated device such as a foot 
or hand switch, or head motion device, or sound or voice recognition system, in a 
preferred embodiment, is selectively permitted to control the override of the normal 
progression of the music's play. 

The override may cause the progression to go backwards or forwards in the music 

1 5 score irrespective of the normal reading of it. The performance mode AutoMode blocks 
the user override to permit performance according to proper material timing and either 
progresses responsive to the music composition data timing, or in an optional 
embodiment, to the performer. This automatically moves through the musical score as 
written and preferably shows an indication of metronome time and an indication of the 

20 proper place in the score where the performer should be for that instrument at any 
specific time. This is especially valuable in a conductor mode of networked 
communication, where a conductor couples to one or more music workstations. 

The user's performance can be compared to the score, and feedback can be 
provided to the performer as to the quality of their performance. 

25 In the performance monitor mode, for a single user or multiple users, the user (or a 

remote teacher or conductor) can indicate the rate at which he feels the performer should 
be performing. A microphone input on the music workstation samples the user's actual 
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performance and permits providing a graphical mapping (for the user or 
teacher/conductor) showing the relative synchronization of the performer's actual 
performance versus the conductor's desired performance. 

In an alternate automatic advanced mode, the display of the music composition is 
5 synchronized to the performers actual performance. Thus, rather than simply indicating 
visually for the teacher/conductor or user what their relative performance was to the 
written displayed musical composition, the relative performer to written music 
synchronization information can be utilized to adjust the display rate of the actual 
musical composition to match that of the performer. 

10 With use of appropriate sound baffling, a plurality of instruments can 

simultaneously be monitored and controlled by the conductor, so long as each 
instrument's output sound pattern is communicated directly to a respective workstation. 
The output of each of the workstations can then be coupled to the conductor's master 
workstation for further analysis and processing. 

1 5 A workstation for an oboe may have a built in slide boom with a featherweight 

microphone to be able to receive sound input from the oboe. Electric instruments, such as 
guitars, keyboards, and other electrical analog signal sources can be fed directly to a line 
input that is appropriately buffered and filtered. Signal input can also be accommodated 
through a MIDI-interface subsystem that permits both utilization of data in workstation 

20 to workstation communications and utilization of MIDI-output at the station where the 
data was input. 

For networked virtual performance, and for one aspect of output display 
presentation, the utilization of MIDI input and MIDI output, at each of the individual 
workstations and at the master controller workstation, permits the capture of the user 
25 performance and conversion to individual performance data which includes time 

synchronization information which can be communicated to the master workstation, 
which synchronizes and combines the individual performance data to generate combined 
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virtual performance data which is then communicated back to the individual workstations 
which utilizing their MIDI output interfaces provide for display presentation ( e.g. audio 
output) of the combined virtual performance data. Additionally, even where virtual 
performance mode is not selected, the provision of MIDI interface input and output on 
5 the workstations have other multiple beneficial users as discussed elsewhere herein. 

Additionally, other types of user performance to user performance data input devices and 
transducers can be utilized, as are well known commercially available including variations 
of analog digital converter input devices, audio signal capture, etc. 

By combining the conductor and performance mode operations, the workstation 

10 can be enhanced to provide training and comparison of performance to actual music. 

Some music is only available in annotated forms where there is not an existing 
signal showing proper synchronization of the signals. Thus, a controller subsystem (such 
as (580)) provides for real time conversion and analysis of syntax of the music notation, 
in conjunction with a precision clock metronome, and provides an indicator (such as color 

15 or other highlighting or holding or accentuating) of the relative timing of the performance 
relative to a place in the sheet music (or other form of musical notation). 

Existing forms of music notation can be converted manually, or can be converted 
automatically by scanning in sheet music, recognizing (using optical character recognition) 
the various elements of the music, and facets and specifics of the syntax in the form of 

20 notation including its constants and variables and protocols, and integrating via an 
artificial intelligence type expert system that notates, highlights, and accentuates via 
synchronized metronoming of time signature to music. Any of a variety of other means 
of converting music can also be used, such as direct input of musical performance signals 
processed via software that converts it into musical notation. Such software is 

25 commercially available, such as from ARS NOVA, Wildcat Canyon Software, Mark of 
the Unicorn, Inc., and Passport Designs, Inc. 
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Since the music notation is now in computer usable form, it is now an easy task to 
communicate, display, compose, alter, and transpose music (such as in key, for types of 
instruments or voice parts, and harmonies) via well known techniques. 

Additionally, where the user input is converted into user performance data for the 
5 workstation, the users individual performance data is also now in computer usable form 
and if appropriately converted from performance to performance data, includes 
appropriate synchronization data relative to master controller performance 
synchronization signal. Thus, both the music notation is in computer usable form 
(making it easy to display, alter and analyze/compare), and the users performance is in 
10 computer usable form (digital individual performance data), it is possible to provide 
intelligent operation and analysis and utilization of both the music composition 
information and the user performance information, to provide for various automated 
modes and features to the users 

Implementation can also be in a custom design comprised of a microprocessor, 
15 non-volatile storage memory, read/write memory, and whatever additional peripheral 

circuitry is needed (such as are available in ASICs, or single chip microcomputer chip sets 
including CPUs, DSPs, A/D, and other peripheral support circuitry). These single or 
multiple chip solutions can be utilized to create a dedicated system to perform complete 
music workstations performance criteria to support an extremely low cost, high volume 
20 music workstation solution. 

A new form of communication is created in that both the process of 
communicating via standard notation is respected and adhered to, while at the same time 
permitting interaction and communication of music media signals ranging from simple 
analog, digitized (analog to digital converted), individual performance data (representative 
25 of the user's performance), and can optionally include timing/synchronization data. 

A multi CD ROM changer accommodates indexed storage of hundreds of 
thousands to millions of musical compositions to permit complete stand alone operation 
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of the user music workstation. Alternatively, an optional built-in or external modem can 
be provided to permit intercommunication with a remote central music database 
management system that permits both communication and down loading (and disconnect) 
for stand alone operation. Thus the workstation can stay on-line, pulling up music as 
5 needed, or can request a single or multiple pieces of musical works be provided to it, that 
are then downloaded from the central database manager. The user workstation then 
disconnects from the music database management system, and thereafter operates stand 
alone where all desired music is stored locally in storage (preferably non-volatile). Storage 
can be semiconductor, magnetic, optical or any other medium. 

10 The use of virtual reality technology, including motion sensors and body gloves, 

permits monitoring of various other things (as shown in FIG. 9). For example, as shown 
in FIG. 10, a camera in conjunction with analysis logic, such as expert software, can 
monitor motion of role model behavior and compare performer behavior. Hand, finger, 
arm, leg, eye, head, body, and mouth movements can all be monitored and constructive 

15 critical feedback can be accumulated, analyzed, and fed back to the user or teacher, for 
performer training, or performances, or for conductor communication. 

The input of monitored movement data is provided to the user workstation, 
permitting precise mechanics training such as finger position, the angle of striking of 
strings relative to the neck of a violin or guitar, or they can be used to permit the virtual 

20 performance of music by a performer using a virtual link apparatus such as a virtual 

reality glove and head movement detection apparatus. The user can then perform a piece 
with their own personalization without any musical instrument in fact. 

For example, the guitar portion for a piece of music could be displayed in notation 
form and actually performed according to the timing of movements of the user's fingers 

25 (either actual fret positions, or only timing information). To add further reality, a mock 
guitar, keyboard, flute, or other instrument can be used and combined with virtual effects 
to provide for music performance and personalization. Thus, for entertainment purposes, 
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users could perform as part of a symphony orchestra playing a violin portion. If they 
performed out of time, they would hear their instrument's performance out of synch with 
the rest of the orchestra's performance. 

There are numerous ways to embody the conductor movement interpretation 
5 system. As illustrated in FIGS. 9 and 10, one is utilizing the body movement detection 
apparatus prevalent in virtual reality, sports medicine, etc., as discussed above, to 
identify specific movement patterns or signal parameters associated with certain 
movement patterns, to initiate a display presentation, audio, video, or audiovisual to 
provide a presentation associated with movement of the conductor. Alternatively, other 

1 0 techniques can be used such as taking the video feed from a video camera or other video 
source (e.g. VCR) and having the conductor interpret his movements and assign them 
unique meanings, to create a lexicon of his movements and corresponding meaning. 

For example, rapid downward movements of the hand from up to down, in a 
certain manner, indicate "decrease the volume " When he points at a particular section at 

1 5 the same time as he is doing that, he is indicating that only that orchestra section is to 
reduce volume. In this manner, either camera input of movements, glove sensing of 
movements, or other techniques (such as audio, ultrasonic, etc.) can be used to track 
movement to permit associated meanings to be attached or indexed to particular signal 
parameters or parametric signals of the meaning of the movement parameters as provided 

20 by the conductor input device. For example, in the case of the virtual reality glove, that 
input would be the signal output of the glove as interpreted by associated software in a 
processor (such as a PC or a MAC). Alternatively, for example, in the case of video 
camera input, it could be pattern recognition or analog signal comparison to determine the 
presence of certain signal patterns indicating to the system to initiate automatic 

25 communication of a conductor presentation. In so doing, the conductor is able to rapidly 
convey his meaning, focus it to a particular group of instruments, and be done with it. He 
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doesn't have to focus very long or concentrate to make sure they've gotten his signal. 
Instead he can focus on listening to see if they got his message. 

FIG. 8 illustrates an alternate embodiment of the present invention. In this 
embodiment, the workstations are remote units (801-803) used by a member of a 
5 marching band. Each of the remote units (801-803) are equipped with receivers (810- 
812) that receive musical compositions transmitted to them. Remote units controllers 
(820-822) control the operation of the remote unit (801-803). The musical composition 
is displayed on the remote unit's displays (830-832) which displays can be an LCD 
multiple line display providing low cost, low power usage, and high visibility/readability, 
10 and with Auto Advance Mode, the display automatically scrolls as the music is to be 
performed. 

Each remote unit (801-803) can be mounted on the instrument on or in place of 
the lyre. The remote unit's antenna (840-842) can be separate from or built into the 
remote unit or the lyre. 

15 A transportable main unit (850) is used to transmit musical compositions to the 

remote units (801-803). The transportable main unit (850) is comprised of a controller 
(806) for controlling the transportable main unit (850), a music database storage medium 
(805) containing the data for the musical compositions to be played by the band, and a 
transmitter (804) for transmitting the musical compositions to the remote units 

20 (801-803), This main unit can be in the form of a suitcase or briefcase size item. The 

main unit can also be provided built into a van that is driven around with the band or as a 
small self-contained portable unit. In accordance with this embodiment, the band can 
play a virtually unlimited number of musical compositions without the problem of 
carrying the music with them in paper form. It also relieves the band members of the 

25 problems of changing music and changing pages while marching. As discussed in the 

above embodiments, in the performance mode, the musical score is automatically scrolled 
across the screen display (830-832). Additionally, a keyboard and/or microphone can be 
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attached to the transportable main unit allowing the conductor to send messages to the 
remote units via displays (830-832) or via a speaker associated with units (801-803). 
This allows the conductor to send instructions to the band (such as to take a certain route, 
or play at different volumes or speeds). With bidirectional communications and user 
5 performance feedback, the conductor can also monitor for errors. 

FIG. 9 illustrates a conductor, stage hand, or other person with a sensor glove on 
each hand (935) and a head and eye movement monitor (930). The figure also illustrates 
the conductor wearing full body sensor equipment (940). Either embodiment or a 
combined embodiment can be used to map body movements. If only the gloves (935) or 

10 body sensors (944) are used, the movement of the glove or sensors can be captured by a 
video system, as illustrated in FIG. 10. 

Other methods that capture motion rely on specialized sensors (944) placed on a 
performer's joints, such as via a sensor body suit (940). Once motion has been filmed or 
analyzed, a data set is produced to interpret that movement into Cartesian coordinates. 

15 These coordinates provide the spatial location of each of those markers. This information 
is then cleaned up and input to an animation package. 

FIG. 10 illustrates a video camera (1005) and a standing conductor (1015) (or 
performing musician to be tracked or virtually linked to perform), with or without a blue 
screen (1010) behind him. The video camera (1005) feeds a video signal to the video 

20 processing system (1020) that utilizes signal processing to provide signal pattern 

recognition capability. The blue in the screen is filtered out in the signal processing such 
as by an Ultimatte process. 

In one embodiment, the conductor is wearing a sensor equipped body suit (940) 
and gloves (935) of FIG. 9. In another embodiment, the conductor is wearing only the 

25 sensor equipped gloves (935) of FIG. 9. In still another embodiment, the conductor's 
movements are picked up by the video camera (1005) and processed without a sensor 
suit. 
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Simple things, like looking for the conductor's rapid hand movements, focusing on 
specific hand movement areas, facial and head movement, arm movements, and body 
language can all be programmed into the recognition knowledge base. Some of the 
technology for complete mapping of body movement that exists in making video games of 
5 today are illustrated in Video Systems magazine, page 42, October 1995, Vol 21, No. 11, 
and NEXT Generation magazine, pages 49-54, October 1995, both incorporated herein by 
reference. 

In any event, having now obtained knowledge related to recognition of the 
movements, the system can interpret them and utilize them to convey presentation 

10 information to the ensemble or orchestra or studio members, or to analyze a performer's 
movements, or to permit a virtual performance. One example would be a large screen 
television or multiple large screen televisions for viewing by the members of the viewing 
group. Alternatively, each music stand could provide for a picture in picture display of 
special movements of the conductor in areas of the display where music is not currently 

15 being played. Since the stand can have the intelligence to compare the performed music to 
the played music, that embodiment permits display of the message in portions of the 
music display area which have already been performed or are not going to be performed 
for some time (e.g., at least ten seconds in either direction; other criteria could 
alternatively be used, and can be set up for desired characteristics of the performing 

20 environment). 

Voice recognition and response to conductor commentary can supplement the 
system. The system could record the message, interpret to whom the conductor directed 
the message and convey it audibly or translate it into a text or icon display as a part of the 
system's audiovisual presentation. 

25 Referring to FIG. 1 1 A, an alternative embodiment of the automated mode "A 

Mode" (240) of FIG. 2C is illustrated. The illustrated process for the automated Mode 
240 of FIG. 1 1 A includes process steps for the Auto Advance A Mode 1, Training A 
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Mode 2, Performance Mode A Mode 3 which are equivalent to the illustrated A Modes 
1, 2, and 3 of FIG. 2C. However, as shown in FIG. 1 1A, at step 1148, the decision is 
made whether the conductor automated mode is selected, and if a "yes" decision, then 
automated Mode 4 is entered at Step 1149, and if a "no" decision, then logic goes to the 
5 networked virtual performance mode A Mode 5, as decided at decision logic step 1150, 
and responsive to the selection of the networked virtual performance mode, A Mode 5 
(1151) is executed. If networked mode A Mode 5 is not selected, then processing 
proceeds to decision logic step 1152 to decide whether the synchronization of the display 
to the performance (Mode A 6) has been selected, and if so, then the processing proceeds 

10 at step 1153 to provide A Mode 6 functions. The automated mode processing ends at 
step 1154. It is to be noted that the automated modes can be cumulatively selected, so 
that the networked virtual performance can include any or all of the other automated 
modes, including A Modes 1, 2, 3, 4, 5, and 6, or other A Modes which may be provided. 
It should also be noted that the tracking of the performance by the user relative to the 

15 presentation of the musical composition (step 272 of FIG. 2D) provides the first part of 
the logic necessary for the A Mode 6 (synchronization of display to performance), as 
illustrated in FIG. 13 in further detail. 

Referring to FIG 1 IB, there is illustrated an alternative overall operation of a 
music composition communications system, alternative to that illustrated in FIG. 2A, 

20 showing additional automated modes including the network virtual performance mode A 
5. FIG. 1 IB illustrates an alternative control logic structure for the network selection, 
where the decision logic proceeds in an analogous manner to FIG. 2A. The operation 
starts at step 1100, and a mode is selected at step 1110. A composition is selected at 
1120, and if the decision logic 1120 determines the composition has been selected, then 

25 processing proceeds at step 1130, which provides for selection of the mode. If no 

composition has been selected, processing proceeds back to the select composition step 
1110. Once the composition has been selected, then a Mode is selected as illustrated at 
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step 1130, and the decision logic of select Mode 1140 determines whether the selected 
mode is that of (1) a display mode, in which case processing proceeds to step 220 of FIG, 
2A, (2) a modified composition mode, in which case processing proceeds as from step 
215 of FIG. 2A going to the yes branch, or (3) the select mode determines that the 
5 networked virtual performance mode has been selected, in which case processing proceeds 
to the networked mode automated Mode 5, step 1150, as illustrated in further detail in 
FIG. 12. 

Referring to FIG. 12, there is illustrated the networked virtual performance 
process flow, step 1150 of FIG. 1 IB, in greater detail. At step 1150, the automated 

10 mode of networked virtual performance has been selected. This automated mode 5 is 
processed by proceeding to verify the selection of the composition. At step 1220, the 
user's performance is captured (e.g., by a microphone or via MIDI) and converted to 
individual workstation performance data. This data is then broadcast by each of the 
plurality of individual music stands to a designated master controller. At step 1230, the 

1 5 controller receives and combines the plurality user's individual performance data to 

generate a combined performance data representative of the synchronized and combined 
plurality of individual performance data. This combined performance data is then 
broadcast to all of the individual music workstations. At step 1240, each of the plurality 
of individual music workstations provides an output (e.g., audio) responsive to the 

20 combined performance data, which is representative of the combination of the plurality of 
users audio performances of the music composition, synchronized to the display 
presentation of the music composition. Timing synchronization data is provided to 
synchronize the plurality of individual music workstations individual performance data to 
one another and to the selected musical composition's display presentation, to facilitate 

25 synchronization for purposes of combination and audio playback at the individual 

workstations, in approximately real time, so as to provide apparent real time feedback of 
the individual user's performance in combination with all other individual users, played 
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back at each of the individual workstations. At step 1250, analysis is done as to whether 
the composition is done. If so, processing completes at step 1260 and ends this 
automated Mode 5. Otherwise, if processing is not done, that is the musical composition 
is not over, then processing proceeds back to step 1220 as illustrated. 
5 Referring to FIG, 13, there is illustrated automated Mode 6 corresponding to 

synchronizing the actual display presentation to the user's performance, providing 
resynchronization of the display of the music to account for the actual performer's 
timing, and any errors or differences. Automated Mode 6 is analogous to A Mode 1 
without override, but not to A Mode 3, which auto-advances independent of the user's 

10 performance. At step 1153, the automated modes 6 is entered. At step 1310, analogous 
to step 272 in FIG. 2D, the user's performance is tracked and stored for comparison to 
the musical composition display presentation relative to the timing of where the 
performance should be. At step 1320, the performance that has been tracked is compared 
to the score and deviation performance data for that performer is generated. At step 

15 1330, the errors are utilized to resynchronize the display presentation of the musical 
composition to adjust to the actual user's real time performance timing relative to the 
display presentation and a synchronization of the commencement of the user's 
performance relative to the musical composition's presentation on the display. At step 
1340, an analysis is done as to whether processing is complete (e.g. is the musical 

20 composition done?). If so, A Mode 6 ends at processing step 1350, and either the 

automated modes of processing can end, or other automated modes can be checked for, 
and processing can begin for another selected automated mode. 

Referring to FIG. 14, there is illustrated the communications architecture and 
system in accordance with the present invention, wherein there are a plurality of 

25 individual music workstations 1410, each responsive to a user's performance of a musical 
input (1411) which is processed by the individual music workstation 1410 to provide a 
performance data output 1412 coupled for communication to a control system 1450, 

1757 appl.v4 (1/26/2000) jig - 43 - 



PATENT APPLICATION 

Atty Dkt No. STD 1757 



which can be a master controller or master workstation, which is coupled for receiving and 
processing the individual workstation performance data for the plurality of individual 
workstations. A communications interface 1413, such as a modem, can provide for 
coupling and communication between the individual workstation 1410 and the master 
5 controller 1450, In one embodiment, the modems 1413 are coupled via a public switched 
telephone network to a corresponding modem 1455 which couples to the master 
workstation to communicate by directionally between the individual workstations and the 
master workstation. Alternatively, any sort of cable, hardwired, or wireless 
communications interface and provided, and would work equally well when in accordance 

10 with the present invention. The master controller 1450 is responsive to the individual 
performance data outputs from the plurality of individual workstations to provide a 
combined and synchronized virtual performance data output 1451 which is coupled via 
the communications network to the plurality of individual workstations, which each then 
respectively provide an output (e.g. audio) 1415 for presentation to the user, 

15 representative of the combined performance data. An audio and/or visual presentation 
output 1452 can also be provided at the master workstation. 

Referring to FIG. 1 5, there is illustrated the data flow and communication 
architecture for one embodiment of the present invention. A music communication 
system is provided that has a plurality of physically separate remotely located locations, 

20 each having one or more individual music workstations, wherein each of the individual 
music workstations provides for communication output of individual performance data 
for a user's performance of a displayed musical composition at that individual 
workstation. As illustrated, a master controller 1505 is coupled to the communications 
infrastructure 1510 at a first location. A plurality of individual music workstations 1511, 

25 1512, 1513, and 1514 are coupled to the communications infrastructure 1510 at location 
1, designated performance group 1. Similarly, location 2, performance group 2, provides 
individual workstations 1521 and 1522 coupled to the communications infrastructure 
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1510, and location 3, performance 3, provides individual music workstation 1531, 1532, 
and 1533 coupled to the communications infrastructure 1510. The plurality of individual 
workstations 1511, 1514, 1421-1522 and 1531-1533, each provide individual 
performance data, providing multiple discrete time samples of performance data (e.g. 
5 time, segments of data), which can include synchronization information therein, which are 
communicated via the communication infrastructure 1510 to the controller or master 
workstation 1505 The resultant output (e.g. audio) from the individual workstations is 
the resynchronized combined performance data of the plurality of individual performance 
data, resynchronized for each of the time segments of output. 

10 Referring to FIG. 16, the data word structure for one embodiment of the present 

invention is illustrated. The data structure for the individual user performance data is 
illustrated as comprising a header comprised of control information including a start of 
word, cyclic redundancy code (CRC) word, the performer's ID (a unique address for the 
individual workstation^ which not only can provide an unique ID but can also provide for 

1 5 indication of an instrument type), and other control information for other modes of 
operation in addition to the virtual performance), payload data representing the actual 
performance data, and timing synchronization data. Any other data word structure 
providing for header information including control information, and payload information 
including performance data, and some timing synchronization scheme, whether or not part 

20 of the user performance data word will work equally well with the present invention. 

FIG. 17 illustrates a networked system embodiment in block diagram form, 
illustrating the network interface subsystem and music data processor subsystems for one 
embodiment of the present invention, compatible with the architecture as described in 
FIGS. 1-15. FIG. 17 illustrates a single individual workstation 1710, although a plurality 

25 of individual workstations 1710 are usually involved in the networked system (analogous 
to the plurality of individual workstations 1410 of FIG. 14). The individual workstation 
1710 provides for communication with a master workstation 1790, (analogous to the 
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communication architecture with the master workstation 1450 of FIG. 14). The musical 
performance by the user (1712) is coupled to an input interface 1715 which provides 
musical performance data to a musical data processor 1720, which can be a MIDI based 
processor, which provides the music processed performance data output to a network 
5 communications interface 1730 which outputs the individual musical performance data 
1731 for coupling via the communications infrastructure to the master workstation 1790. 

The master workstation 1790 processes the individual musical performance data 
1731, from the plurality of individual workstations 1710, and provides an output 1791 of 
combined virtual performance data, wherein both the individual musical performance data 

10 and the combined virtual performance data are comprised of a plurality of discrete time 
segmented data samples, wherein the combined virtual performance data is synchronized 
within each sample time to provide for synchronized combination of the individual 
musical performance for that time segment. 

A network interface subsystem 1735 is responsive to the combined virtual 

15 performance data to provide synchronized virtual performance data output to the music 
data processor 1725 (e.g. MIDI) which provides a sound output of the combined virtual 
performance data corresponding to the synchronized combination of the individual 
musical performance data from a plurality of the individual workstations 1710. 
Additionally, the individual workstation 1710 provides data analysis and presentation 

20 processing 1740 which provides an output responsive to, inter alia, the combined virtual 
performance data to a presentation means such as an audiovisual display, or a video 
display or an audio output, 1745. Note that in a preferred embodiment, the multiple 
individual network workstations 1710 are synchronized (e.g. responsive to 
synchronization signal from the master workstation 1790), so that the plurality of the 

25 individual workstations simultaneously commence display presentation of the musical 
composition for synchronization of individual user performances. In one embodiment, 
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the output of the individual performance data (such as via a MIDI music processor) is 
comprised of timing synchronization information that is used by the master workstation 
to synchronize all of the musical performances representative individual musical 
performance data to virtually perform together in a combined and time-synchronized 
5 performance. The combined virtual performance data representing the synchronized 
virtual performance is then sent back (e.g. broadcast) to the individual workstations, 
coupled via the network communication interface 1735 back to the individual 
workstation's music data processor 1725, to provide audio and/or visual output 
corresponding thereto. Where the master workstation is for a conductor or teacher, the 

10 conductor mode or teacher mode can also be utilized in addition to the network virtual 
performance mod. In fact, all of the automated modes can be selectively combined, such 
as by providing layering of functions via software. 

Referring to FIG. 18, the timing diagrams for the individual performance data and 
synchronized virtual performance data is illustrated, relative to signals output from the 

1 5 individual networked workstations NWSj, NWS 2 , NWS N and a master workstation 
MWS. As illustrated, the master workstation MWS provides a synch signal to 
commence synchronization of all workstations (NWS) coupled to the system. In a 
preferred embodiment, the individual workstations (NWS) begin the display presentation 
of the musical composition (as does the master workstation) in synchronization with the 

20 synch signal from the master workstation. Responsive thereto, each of the individual 
workstations displays its musical composition, and the users of the workstations can 
perform the musical composition. Individual workstations (NWS) provide individual 
performance data responsive to the user's performance. As illustrated, workstation 
NWS! provides performance data A and B at respective times, while in the individual 

25 workstation NWS 2 provides individual performance data C & D at respective times, and 
independent workstation NWS N provides individual performance data signals E&F at 
respective times. During the first time segment, individual performance data A, C & E are 
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output, respectively, from individual workstations 1, 2, and N. However, as shown in 
FIG. 18, the relative timing of output of the individual performance data (and receipt by 
the master workstation thereof) are skewed from one another in timing. The master 
workstation resynchronizes the individual performance data A, C, and E within each time 
5 segment, and provides an output of the recombined reconstructed and synchronized A, C, 
and E to provide synchronized combined virtual performance data. It is to be noted that 
synchronized virtual performance data A, C, E occurs within the first time segment Tl, 
which contains both the individual performance data and the corresponding responsive 
synchronized virtual performance data. The maximum time delay T D represents the 

10 acceptable worst case time that a user can tolerate between the user's performance output 
of corresponding individual performance data and the time until the synchronized virtual 
performance data corresponding to that individual performance data is received by the 
same individual workstation and provided as audio output. Similarly, during time slot 2, 
each of the individual workstations provides individual performance data B, D and F, 

1 5 slightly time skewed from each other, and the master workstation provides combined 
synchronized virtual performance data B, D, F, which is communicated back to the 
individual workstations to generate an audio output. 

The present invention finds application in many different environments and 
embodiments. For example, a live practice can be held from multiple locations, while one 

20 or more musicians are editing the score and communicating the changes real time, re- 
performing them, etc. An orchestra can be at one location, while a singer can be at another 
location. Creating and editing and synchronizing music to movies scores can also be 
facilitated with the present invention with beneficial gains. For example, in a movie, a lot 
of elements are changing dynamically that need to be resynchronized. For example, a 

25 scene can be lengthened or shortened, and the soundtrack must be adjusted. A score 
which was created to the movie must be rewritten to accommodate the edited movie. 
Typically, the composer creates the score at home or at a studio. That score is sent to 
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the movie studio. However, the movie studio may edit the scene that music is used in, 
say a reduction of three seconds in the scene. The edited scene is sent back to the 
composer so that he can shorten his music. The process is slow, inefficient, and an 
impediment to the production process. By utilizing the present invention, this can be 
5 accommodated in real time, and from remote locations, and the virtual performance mode 
as well as the other modes of the music communication of the present system can be 
utilized advantageously for this application. 

The present system provides an infrastructure for a music Internet. It can be 
applied to interactive editing of music. It can provide for special one on one mode for 

10 teaching and performance evaluation. It can facilitate adapting a music display to the 
performer's performance, or drive the display to a metronome and force the user's 
performance to be synched to the metronome timing. Music can be composed, edited, 
transposed, etc. easily, from any of the plurality of individual workstations or on a single 
standalone workstation. Multiple individual music workstations (IMW's) can 

1 5 communicate with one another, both with data and audio such as voice and/or music, 
and/or supplemental data such as video, audio, data files, control signals, applets, etc. 
Bands can practice at home without the disadvantage of either practicing alone or having 
to physically get together in a large grouping Singers can group with other instruments 
and other vocalists via respective ones of a plurality of IMW's, one, two, or even the 

20 whole plurality being remotely located physically from one another (e.g., LA, NY). 

There can be multiple instruments playing at a single location. There are many 
options regarding how IMW's are allocatable, such as. (1) on IMW (Individual Music 
Workstation) per location, (2) IMW per instrument type, (3) one IMW per performer 
(e.g. Instrument, vocal). There are also many options (1) for networked communications 

25 between the IMW's and/or between the IMW's and the Master Workstation. Modes can 
be selectively mixed and matched and combined, such as (1) all auto-modes enabled mode 
(2) virtual performance mode, full audiovisual, (3) audio link only mode, etc. There are 
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also many options as to modes of signal processing which can be applied to the user 
input to generate the individual performance data, ranging from simple analog to digital 
(A/D), to complex musical sound analysis algorithms (e.g. to detect characteristics such as 
instrument type, pitch tone, attack, decay, amplitude, voice signatures, etc.) There are 
5 also tremendous numbers/parameters of performance analysis algorithms that can be 

selected and which utilize the selected musical composition data (corresponding to that 
displayed on the IMW which is receiving the performance input for analysis) (e.g. the 
sound characteristics generated by the sound analysis of the user's performance, to (3) in 
between "(1)" and "(2)". Additionally, many kinds of data can be communicated between 

10 (1) IMW's (2) IMW's and a Master Workstation, and (3) IMW's and external 

communication interfaces, etc. Examples of the kinds of data include performance data, 
video data, processed/analyzed performance data, audio signals, data files, etc. 

Referring to FIG. 19, the memory work space structure for both the individual and 
the master workstations, in the virtual performance mode, is illustrated. Each of the 

15 individual workstations provides for music data buffering o the outgoing live performance 
data, individual performance data IPDXXX, and further provides for storage of the 
incoming combined virtual performance data CPDXXX. This buffering is done within 
read-write memory within the individual IMW's. Similarly, the data structure for the 
memory buffer of the master workstation is illustrated, showing separate buffer storage 

20 locations for each of the plurality of individual workstations WS! to WS N , providing 
storage of associated respective individual performance data IPD A-N. In one 
embodiment, multiple frame buffers are provided in the IMWs, so that multiple time 
segments of IPD and CPD data can be stored in the buffer structure, and so that the 
buffer structure can support both input storage and output retrieval operations. Other 

25 data structures and buffering operations are also compatible with the present invention. 
The structure of the data word can be changed or adapted within its existing structure to 
facilitate the transfer of supplemental data is transferred from the IMWS or Master 
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Workstation, or between the IMW's, or between the Imws and the Master Workstation. 
By utilizing the illustrated or other compatible data buffer structure, timing and 
synchronized (with the frame buffered variety) asynchronous communication are 
facilitated, not only as between communication workstations, but furthermore, as to the 
5 operations internally of the IMW. With frame buffering, the individual workstation and 
master workstation can dump the result output to the buffer, or receive input individual 
performance data asynchronously and buffered, relative to the other operations of the 
respective workstation, The IMW can facilitate operation in one mode or in multiple 
modes concurrently. 

10 From the foregoing, it will be observed that numerous variations and modifications 

may be effected without departing from the spirit and scope of the invention. It is to be 
understood that no limitation with respect to the specific apparatus illustrated herein is 
intended or should be inferred. It is, of course, intended to cover by the appended claims 
all such modifications as fall within the scope of the claims. 



1757 appl.v4 (1/26/2000) jig 



-51- 



PATENT APPLICATION 

Atty Dkt. No. STD 1757 



WHAT IS CLAIMED IS: 

1. A music display system for use by a plurality of users in providing a 
plurality of display presentations of a selected musical composition, said system 
comprising: 

5 a plurality of individual workstations, each workstation comprising 

a communication interface providing for communications with the 
respective workstation of music data representative of the selected musical 
composition; 

memory for locally storing the data responsive to the 
1 0 communications interface; and 

a display apparatus for providing a local visual display 
presentation representative of the selected musical composition responsive to the 
stored data; 

wherein the system is further comprised of: 
15 means for synchronizing the presentation on the plurality of local 

visual display presentations of the selected musical composition. 



2. The system as in claim 1, further comprising: 

an input device responsive to a performance by the user of the displayed 
20 musical composition for providing an output of user performance data. 



3. The system as in claim 2, wherein the system provides for a display 
presentation of the differences between the displayed composition data and the user 
performance data for at least one of the individual workstations. 

25 
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4. The system as in claim 1, wherein the means for synchronizing is 
responsive to at least one of an embedded timing signal within the music data, timing data, 
and an external timing signal. 

5. The system as in claim 2, further comprising: 

combining means for synchronizing and combining the user performance 
data from a plurality of the individual workstations to generate composite virtual 
performance data, responsive to the user performance data output from each of the 
plurality of individual workstations. 

6. The system as in claim 1, further comprising: 

an editing subsystem for changing features of at least one of pitch, key, 
tempo, instrument type, notation, and composition of the music data to create modified 
music data; 

wherein the modified music data is communicated to at least one of the 
individual workstations which provides a local video presentation responsive to the 
modified music data. 

7. The system as in claim 6, wherein the modified music data is distributed to 
a plurality of the individual workstations, which in a synchronized manner each provide a 
local video presentation responsive to the modified music data. 

8. The system as in claim 6, wherein the editing subsystem is a part of at 
least one of the individual workstations. 

9. The system as in claim 6, wherein the changing is restricted to permit 
changing of only some of the features. 
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10. The system as in claim 9, wherein the changing of features is restricted at a 
defined level of permission. 

1 1 . The system as in claim 9, wherein there are a plurality of the editing 
5 subsystems; and 

wherein for each of the editing subsystems the changing of features is 
programmably restricted at a defined level of permission. 

12. The system as in claim 7, wherein the plurality of individual workstations 
10 are each associated into defined subsets of individual workstations; 

wherein each of the editing subsystems is associated with at least one of 
the defined subsets and communicates the respective modified music data to the 
respective associated defined subset of individual workstations each of which provides a 
respective local display presentation responsive to the respective modified music data. 

15 

13. The system as in claim 12, wherein at least one of the editing subsystems 
is a master that communicates its respective modified music data to all of the plurality of 
individual workstations. 

20 14. The system as in claim 13, wherein the modified music data from the 

master is given priority for display on the individual workstations over all the modified 
music data from all other ones of the editing subsystems. 

15. The system as in claim 1, wherein the communication between the 
25 individual workstations is bidirectional. 
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16. The system as in claim 6, wherein the changing is responsive to a user 

input. 

17. The system as in claim 16, wherein the user input is at least one of an 

5 audio stimulus, digital data, a switch, a touch input device, motion sensor, motion capture 
data, and speech recognition. 

18. The system as in claim 6, wherein there are a plurality of the editing 
subsystems; 

10 wherein the plurality of individual workstations are each associated into 

defined subsets of individual workstations; and 

wherein each of the editing subsystems is associated with selected ones of 

the defined subsets, wherein each of the editing subsystems communicates its respective 

modified music data to the respective associated selected ones of the defined subsets of 
1 5 individual workstations each of which provides a respective local display presentation 

responsive to the respective modified music data. 

19. The system as in claim 18, wherein at least one of the editing subsystems 
is a master that communicates its respective modified music data to all of the plurality of 

20 individual workstations. 

20. The system as in claim 19, wherein the modified music data from the 
master is given priority for display by all of the individual workstation relative to any and 
all other modified music data from all other ones of the editing subsystems. 

25 
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21. The system as in claim 18, wherein at least one of the editing subsystems 
is a subgroup master that communicates the respective modified music data to the 
respective associated selected ones of the defined subsets of the individual workstations. 

22. The system as in claim 21, wherein there are a plurality of subgroup 
masters. 

23. The system as in claim 21, wherein at least one of the editing subsystems 
is a master that communicates its respective modified music data to all of the plurality of 
individual workstations. 

24. The system as in claim 23, wherein the modified music data from the 
master is given priority for display by the individual workstations over all the modified 
music data from all other ones of the editing subsystems. 

25. The system as in claim 23, 

wherein the master is for use by at least one of a conductor, band leader, 
teacher, librarian, and composer; and 

wherein each of the plurality of subgroup masters is for use by at least one 
of a section leader, a band leader, a teacher, and a librarian. 

26. The system as in claim 1, wherein the music data is further comprised of 
instrument type data; 

wherein at least one of the individual workstations is programmed to 
selectively receive the communicated music data responsive to the instrument type data. 
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27. The system as in claim 26, wherein there are a plurality of the individual 
workstations, each programmed to selectively receive the communicated music data 
responsive to the instrument type data. 

5 28. The system as in claim 26, wherein the music workstation is programmed 

to respond to a specific instrument type responsive to at least one of preprogramming, a 
switch, an audio input, a direct line input, MIDI data, user programming, and remote 
program control. 



10 29. The system as in claim 1, wherein the music data is further comprised of 

instrument type data; 

wherein the music data is broadcast to a plurality of the individual 
workstations, each of which provides a local video display presentation responsive to 
processing of the music data to locally convert the music data to customize the video 

15 display presentation in accordance with the respective instrument type data. 

30. The system as in claim 29, wherein the music workstation is programmed 
to respond to a specific instrument type responsive to at least one of preprogramming, a 
switch, an audio input, a direct line input, MIDI data, user programming, and remote 
20 program control. 



3 1 . The system as in claim 27, wherein each of the individual workstations has 
an associated instrument type; 

wherein each of the individual workstations is further comprised of a 
25 receiver that provides communication that is addressably selective to each of the 
individual workstations responsive to the instrument type data and the associated 
instrument type. 
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32. The system as in claim 31, wherein the communication is selectively 
addressable to subgroups within the plurality of individual workstations providing 
frequency band-based communications which is mapped between each of the respective 
bands and each of the subgroups. 

5 

33. The system as in claim 1, further comprising: 

a conductor workstation providing controlled addressable communications 
of the music data to at least one of individual ones of the plurality of individual 
workstations, 

10 

34. The system as in claim 33, wherein the communicating is selectably 
addressable to defined subgroups within the plurality of individual workstations 
providing band-based communications; 

wherein communications is mapped between each of the respective bands 
1 5 and each of the subgroups. 



35. The system as in claim 1, wherein at least one of the individual 
workstations is operable in a user selected automated mode comprising at least one of 
auto-advance mode, training mode, performance mode, auto-repeat mode, conductor 
20 mode, marching band mode, auto-compose mode, self-learn mode, and user activated 
display page turning mode. 



36. The system as in claim 1, wherein one of the individual workstations is a 
master workstation in communication with the remaining ones of the individual 
25 workstations. 
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37. A method of processing and displaying music comprising: 

retrieving stored music data responsive to user selection from a listing of 
available music compositions; 

processing the stored music data to format the music data for presentation; 
displaying a video presentation of the music responsive to the processing. 



10 



38. The method as in claim 37, wherein the displaying is further comprised of: 
displaying, on a plurality of separate display apparatus, the video 

presentation of the music, responsive to the processing. 

39. The method as in claim 38, further comprising: 

distributing the processing and displaying among the plurality of display 

apparatus. 



15 40. The method as in claim 37, further comprising: 

changing the music data as to at least one of key, notation, display format, 
instrument type, and mode, to provide modified music data; 

wherein the processing provides processing of the modified music data. 

20 41 . A communications system comprising: 

a plurality of individual music workstations, each comprising a music 
input for selectively providing respective individual performance data output, responsive 
to a performance by a user of that respective individual music subsystem; 

combining means, responsive to the individual performance data output 
25 from each of the plurality of individual music workstations, to provide a combined output 
of composite virtual performance data; 
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wherein the combining means is further comprised of means for 
synchronizing and combining the individual performance data from the plurality of 
individual music subsystems to generate the composite virtual performance data; and 

means for communicating said composite virtual performance data to at 
5 least one of the plurality of individual music subsystems, which provides a local 
presentation representative of the combined musical performances for all of the 
communicating plurality of individual music subsystems responsive to the composite 
virtual performance data. 



10 42. The system as in claim 41, 

wherein each of the individual music workstations is further comprised of 
a music display apparatus for providing a local visual display presentation of a selected 
musical composition; 

wherein the plurality of individual music workstations provide for 
15 synchronized display presentation of the musical composition. 

43. The system as in claim 42, wherein a plurality of the individual music 
subsystems each provide for output of individual performance data representative of the 
musical performance by the user corresponding to the display presentation. 

20 

44. The system as in claim 42, further comprising: 

synchronization means for generating a synchronization signal for start of 

performance; 

wherein the selected musical composition is performed over a time period 
25 and is communicated in discrete time segments, wherein each of the time segments is 
synchronized responsive to the synchronization signal and the individual performance 
data output. 
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45. The system as in claim 44, wherein the combining means provides the 
synchronization signal. 

46. The system as in claim 41, wherein the composite virtual performance data 
5 is communicated back to a plurality of the individual music subsystems. 

47. The system as in claim 41, wherein at least one of the individual music 
workstations provide at least one of an audio output and a visual presentation, responsive 
to the composite virtual performance data. 

10 

48. The system as in claim 4 1 , wherein each of the individual music 
workstations is further comprised of a network interface subsystem. 

49. The system as in claim 42, further comprising: 

1 5 operational selection means for determining a selected operating mode for 

controlling progression of the video presentation. 

50. The system as in claim 42, further comprising means responsive to the 
composite virtual performance data to generate a video presentation. 

20 

5 1 . The system as in claim 41, wherein the user performance data is comprised 
of at least one of audible performance data, visual performance data, electrical signals, 
digital data and control data. 

25 52. A method for music virtual performance, for integrating simultaneous 

musical performances from a plurality of locations of music display workstations into a 
cohesive whole, the method comprising: 
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accepting performance data from each of the plurality of music display 

workstations; 

processing the live performance data into discrete time samples; 
communicating the discrete time samples; 

synchronizing the discrete time samples communicated from each of the 
plurality of music display workstations to provide synchronized communicated time 
samples; 

combining the synchronized communication time samples into combined 
virtual performance data; and 

communicating the combined virtual performance data to provide at least 
one of an audio and a video presentation responsive to the combined virtual performance 
data. 



53 . The method as in claim 52, further comprising: 

providing the presentation on at least one of the plurality of music display 

workstations. 



54. The method as in claim 52, wherein the synchronizing is further comprised 

of: 

20 providing a common time reference signal; and 

utilizing the common time reference signal to synchronize the discrete time 
samples from each of the plurality of music display workstations. 



55. A method of providing a display presentation of a selected musical 
25 composition on an individual music workstation (IMW), comprising: 

communicating musical composition data corresponding to the selected 
musical composition; 
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processing and locally storing the musical composition data; and 
providing a visual display presentation of the selected musical 
composition responsive to the processing and the stored musical composition data. 

5 56. The method as in claim 55, further comprising: 

associating an instrument type to the IMW; 

broadcasting musical display data for multiple separate graphical display 
presentations corresponding to multiple separate respective multiple instrument types; 

discriminating between the multiple separate graphical presentations to 
10 select a specific one representative of the corresponding respective one of the instrument 
types, responsive to the associating and the discriminating. 

57. The method as in claim 56, further comprising; 

providing a video display for the associated respective instrument type 
1 5 responsive to the discriminating. 

58. The method as in claim 56, wherein there are a plurality of IMW's, each 
having an associated instrument type, the method further comprising: 

providing a video display on each of the respective IMW's associated with 
20 the selected instrument type that is associated with the respective IMW. 

59. The method as in claim 54, further comprising: 

providing a source of secondary video data representative of a secondary 

video image; 

25 displaying the secondary video image as a picture-in-picture within a 

subpart of the visual display presentation. 
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60. The method as in claim 54, further comprised of; 
providing a plurality of separate ones of the individual music 

workstations, each of which provides for selective local displaying. 

61. A music display system comprising: 

at least one music workstation comprising means for displaying a music 
composition responsive to music data; 

at least one editing subsystem for changing at least one of the pitch, key, 
tempo, instrument type, notation, and composition of the music data to create modified 
music data; 

means for distributing the modified music data to the at least one of the 
music workstations; 

wherein the music workstation provides the display presentation 
responsive to the distributed modified music data. 

62. The system as in claim 61, wherein the changing of features is restricted at 
a defined level of permission. 

63. The system as in claim 61, wherein there are a plurality of the music 
workstations, and wherein the modified music data is distributed to the plurality of the 
music workstations which each provide a local video presentation responsive to the 
modified music data. 

64. The system as in claim 63, wherein the local video presentations provided 
on the plurality of music workstations are synchronized together. 
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65. The system as in claim 61, wherein the changing is responsive to a user 
input, wherein the user input is at least one of audio, data, a switch, a touch input device, 
a motion sensor, and speech recognition. 

5 66. A method of electronically displaying a music selection on at least one 

display subsystem, the method comprising: 

storing music data representative of a display presentation for the music 

selection; 

communicating the music data to the at least one display subsystem; 
1 0 processing the communicated music data for display; 

displaying a video presentation of the music selection on the at least one 
display subsystem, responsive to the processing of the communicated music data. 

67. The method as in claim 66, wherein there are a plurality of the display 
15 subsystems, the method further comprising; 

communicating the music data to a plurality of the display subsystems; 

and 

displaying a video presentation on all of the plurality of the display 
subsystems of the music selection responsive to the music data. 

20 

68. The method as in claim 66, further comprising: 
modifying the stored music data; 

communicating the modified music data to the at least one of the display 
subsystems; and 

25 displaying a representation of the modified stored music data on the at 

least one of the display subsystems. 
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69. The method as in claim 67, the method further comprising: 

associating a type with each of the plurality of the display subsystems, 

wherein there are multiple different versions of the modified music data; 

selectively providing a video display for each of the plurality of display 

subsystems, responsive to the associated type for the respective one of the display 

subsystems. 
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ABSTRACT OF THE DISCLOSURE 

A musical workstation system produces a display presentation in one of a musical 
composition responsive to musical composition data and responsive to one or both of 
input variables and a selected operating mode. The system is comprised of (1) means to 

5 provide the musical composition data (such as local storage (ROM, RAM, CD-ROM, 
hard disk etc.), or via a communications interface to an external device (such as another 
music workstation, a master controller, a computer), a memory, a selection subsystem, a 
controller, and a display subsystem. The memory selectively stores the received original 
musical compositions. The selection subsystem determines a selected operating mode 

0 and display format. The controller, responsive to the selection subsystem, provides 

means for selectively controlling the storing of the musical composition data in memory 
and selectively processing (e.g. altering) the stored musical composition data responsive 
to the selected operating mode and the input variables to produce a particular one of a 
plurality of processed results, such as external communications, operating, mode, 

5 transformation to derivative musical compositions, etc. The music workstation can 
communicate with one or more external devices, such as other music workstations, a 
master workstation, a controller, etc.. The display system provides for selection of 
original compositions, creation of derivative compositions, distribution of compositions, 
monitoring of each performer's performance, group virtual performances, and also allows 

0 for local and distributed retrieval and editing of musical compositions, such as changing 
keys, pitch, tempo, and many other parameters. The music transformation can be 
performed locally or at the central or distributed music database, which stores the scores 
of musical composition. The musical composition data can be transposed via a controller, 
and can be transmitted to a plurality of the individual music workstations that then 

5 display the music composition, to permit playing by musical instruments or vocalists, 
and for location presentation. 
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